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Optimized Compressed Sensing via Incoherent 
Frames Designed by Convex Optimization 

Cristian Rusu* and Nuria Gonzalez-Prelcic 


Abstract 

The construction of highly incoherent frames, sequences of vectors placed on the unit hyper sphere of a finite 
dimensional Hilbert space with low correlation between them, has proven very difficult. Algorithms proposed in 
the past have focused in minimizing the absolute value off-diagonal entries of the Gram matrix of these structures. 
Recently, a method based on convex optimization that operates directly on the vectors of the frame has been shown 
to produce promising results. This paper gives a detailed analysis of the optimization problem at the heart of this 
approach and, based on these insights, proposes a new method that substantially outperforms the initial approach 
and all current methods in the literature for all types of frames, with low and high redundancy. We give extensive 
experimental results that show the effectiveness of the proposed method and its application to optimized compressed 
sensing. 


Index Terms 

Grassmannian frames, equiangular tight frames, incoherent frames, overcomplete dictionaries, compressed sensing. 


I. Introduction 

Frames m, 0, El are crucial mathematical objects used for the study of overcomplete bases and subsequent ap¬ 
plications. Although they were introduced in the mathematical literature from some time, they have been introduced 
relatively recently to the signal processing community where their capability to represent signals in overcomplete 
bases is exploited. 

The properties of some classes of frames make them well suited in the area of sparse representations and 
compressive sensing 0. For the sparse recovery algorithms, like basis pursuit (BP) ||6l and matching pursuit (OMP) 
Q, 0, a theoretical framework has been developed where the performance of these algorithms can be studied 
using concepts like the mutual coherence 0 and the restricted isometry property (RIP) ifTOl . The mutual coherence 
of a matrix is defined as the maximum absolute value of the normalized dot products between different columns 
and it is thus very easy to compute, while the RIP characterizes how close a matrix is to an orthogonal one when 
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operating on sparse vectors. The guarantees for successful recovery of the sparsest solutions to overcomplete systems 
of equations depend directly on these properties, i.e., frames with low mutual coherence are highly desirable. In 
terms of the mutual coherence the main research direction is to find structures as incoherent as possible (like 
Equiangular Tight Frames mina, while for the RIP classes of matrices that meet desirable bounds have been 
found (we mention random Gaussian and Bernoulli matrices) and used with great success in compressive sensing. 
Furthermore, even though highly incoherent frames have not been extensively used in signal processing they are of 
great importance in coding theory and communications im for the construction of erasure-robust codes d, na, 
m and for the development of spherical codes uni mi- Strong connections have been shown to exist between 
Grassmannian packings, graph theory and spherical codes ED- Applications have also seen frames used in quantum 
computing EOl . 

Thus, the construction of highly incoherent frames is of central importance. 

A. Contribution 

This paper is concerned with the design of real frames as incoherent as possible in an m dimensional Hilbert space 
with N vectors. We propose an algorithm based on convex optimization called Sequential Iterative Decorrelation 
by Convex Optimization (SIDCO). We provide insights into the way it behaves, we are able to give conditions 
under which the algorithm converges, what classes of frames are its minimum and we show experimentally that 
it substantially outperforms all other methods in the literature. SIDCO is a major improvement of a previously 
introduced method for incoherent frame design based on convex optimization Il2n . With the approach of SIDCO, 
the problem lends itself to detailed analysis allowing for the selection of the optimal internal parameters, which in 
El are chosen heuristically, while the solutions achieve lower coherence in shorter running times. 

The application to optimized compressed sensing ll22l . Il23l shows that SIDCO outperforms all previous methods. 

B. Previous work 

In the case of frame design, probably the most influential work consists of a design procedure based on the 
alternating projection method ll24l with an implementation presented in ll25l . These methods concentrate on the 
Gram matrix of the real frame F G since we need to minimize its largest off-diagonal absolute value 

entry. The procedures can be summarized by the following iterative procedure: construct the Gram matrix G of 
the current dictionary, decrease the values of a fixed number of entries in G by a given amount, use the singular 
value decomposition (SVD) to reduce the rank of G to m and then factor out the new dictionary. Besides the need 
to run for a large number of iterations, this class of methods proves to work well in practice, especially for small 
sized dictionaries. Still, there are some internal parameters that need to be tuned: the number of entries of the Gram 
matrix to be updated and how big this update is. Previous to this family of methods, in the field of communications, 
algorithms were developed to construct tight frames, that are the optimal sequences for DS-CDMA systems, by 
minimizing the total squared correlation (TSC) ll26l . Related work on minimizing TSC for binary sequences is also 
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found in Eaimi. Other methods, like the one in 1291 approach a unit norm tight frame by solving a differential 
equation, while the work in 1^ uses a gradient based method to decrease the coherence of the frame. 

Considerable effort was allocated to the creation of explicit (and sometimes equiangular) frames using con¬ 
structions by conference matrices isa, Kerdock codes l3^ . strongly regular graphs 13^ . Steiner systems l34l or 
adjacency matrices of Paley tournaments l35l . Although in this paper we study real valued frames, extensive work 
studies complex l3^ and complex harmonic l3^ frames. For example, in the quantum information theory literature 
we can hnd constructions of symmetric, informationally complete, positive operator valued measures (SIC-POVM) 
1^ that achieve low coherence for N = m? vectors in complex Hilbert spaces. These frames have been constructed, 
by numerical methods, for all dimensions m < 67 CTl and it is conjectured that they exist for all m. 

Recently, a method based on averaged projections ll40ll has been proposed where the authors improve the shrinkage 
operation proposed from ll^ to produce less coherent frames. This method is the state of the art in incoherent 
frame design. 

The method that we analyze, and improve upon here, was the hrst to treat the problem directly on the frame, and 
not on its Gram matrix. The method EH solves the original non-convex problem of designing incoherent frames 
by considering a series of convex problems such that each reduces the coherence of the previous frame. 


C. Outline of paper 

The paper is organized as follows: Section outlines properties of frames, introduces the optimization problem 


central to our proposed method and its properties. Section III presents the proposed method for incoherent frame 
design. Section |IV] gives extensive experimental results for the proposed method, with comparisons, and Section [V| 
concludes the paper. 


II. Design of incoherent frames 

In this section we present and analyze the convex optimization problem at the heart of the incoherent frame 
design method that follows this section. 


A. Frames 


We call a hnite frame over the held 


a sequence of N vectors fi £ M"*, 1 < * < that satishes 

N 

a\\v\\l<Y,\flvf<l3\\v\\lyv£R^, 


( 1 ) 


and we introduce the frame synthesis matrix F £ consisting of the concatenated vectors of the frame 


F = 


fl f2 ■■■ f. 


N 


( 2 ) 


Constants a, /3 G K, with 0 < a < /3 < oo, are called the lower and upper bounds of the frame. When a = (3 the 
frame is called a-tight and when a = /? = 1 the frame is called a Parseval frame. In this paper, we consider all 
vectors in the frame to have unit £2 norm (i. e. placed on the unit m-sphere), which means that the dot products 





4 


are equal to the cosine of the acute angle between each two vectors. These are called unit norm frames and they 
are the focus of this paper. 

We recall the Gram matrix of the full rank frame to be G = F'^F G . This matrix is symmetric and 

positive semidehnite with rank m, it has a unit diagonal (because of the normalized vectors of the frame) and the 
off-diagonal entries are equal to the dot products between any two distinct vectors. Based on this matrix we dehne 
the mutual coherence as 

= , max (3) 

From its dehnition it is clear that the mutual coherence ^ takes values between 0 and 1, with the lower bound 
reached for orthonormal frames and the upper bound reached when two vectors of the frame are collinear. We dehne 
frames that have a low mutual coherence value to be incoherent. The mutual coherence is useful, for example, in the 
context of sparse representations since it provides a bound on the performance of sparse approximation algorithms 
ED: considering the system Fx = y, with F full rank m, the solution x of BP or OMP is the sparsest if 

The frame potential ll42ll 

N N 

FP(F)=^^|p,,|2 = ||G||^, (5) 

i=l j=l 

was introduced to describe how close the frame is to a unitary tight frame, when FP achieves its minimum of 
/m. 

Since the properties of interest of these frames revolve around the Gram matrix, most design procedures consider 
the G matrix the central object while their constructions are based on the following theorem presented in ll24l and 
the numerical procedure of polar decomposition to recover the frame. 

Theorem 2 of ||24]| . Let F he a m X N matrix with singular value decomposition IfEV^ . With respect to the 
Frobenius norm, the closest a-tight frame to F is given by aUV^. ■ 

We next dehne the Equiangular Tight Frames (ETFs) ED as the sequence of N vectors fj that obey: 


ll/illi = 1 and l/f/jl =/X, 1 <i < j < N,i ^ j (6) 

(7) 

with an immediate consequence of (|^ and Q that the distances between all the different member vectors are the 
same and equal, in absolute value, to 


F = 


I N — m 
m(N — 1) 


( 8 ) 


This relation describes in fact a general lower bound on the mutual coherence that is also known as the Welch 
bound (WB) or Ranking bound ED, ED which we use as a reference point performance, for perspective. This is 
the optimal bound of coherence for N < m?, the regime that we consider here. 

The ratio p = N/m dehnes the redundancy of the frame and we call frames (m, km) to be fc-overcomplete. 
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It is trivial to observe that orthonormal frames are ETFs and actually reach the lowest possible coherence, zero. 
Sadly, we are interested in highly overcomplete frames, i.e., N m, where ETFs only exist for a few pairs (to, N) 
(for example, for the real case, these do not exist if > to(to + l)/2 IE)). When they exist, ETFs provide the 
lowest possible mutual coherence and they are considered the generalization of orthonormal bases because they 
preserve (a generalization of) Parseval’s identity Q. 

A generalization of ETFs is embodied by Grassmannian Frames (GFs). For these structures, the largest absolute 
value dot product is minimum for the given dimensions to and TV. Notice that this relaxed condition means that 
GFs exist for every pair {m,N). Still, this does not help in decreasing the computational complexity of hnding 
them or even deciding if a given frame is a GF. 

From this brief outline it should be clear that creating maximally incoherent frames (either ETF or Grassmanian) 
is an extremely difficult task in the general case and there are no guarantees that the lower bound of coherence can 
be reached for a given pair (to, N). Necessary conditions for the existence of ETFs are given in ifT^ . Il45l . 


B. Incoherent frame design 

This subsection is dedicated to the description of a numerical procedure for the design of incoherent frames. We 
develop an optimization procedure that directly designs the frame F. 

Ideally, we would like to solve exactly the following non-convex optimization problem; 

minimize max | ff fi \ 

F ^,r, jV* (9) 

subject to ||/i ||2 = 1; 1 < i < N. 

Solving this problem exactly would produce ETFs, when they exist, or Grassmanian frames otherwise. The difficulty 
lies in the fact that both the objective and the equality constraints are non-convex. We proceed to relax this problem 
and provide convex optimization formulations to solve it approximately. 


III. The proposed algorithm 


The approach taken in this paper is not to solve (|^ directly but given a frame H = 


hi 


h 


N 


-nmxN 


fi 


f. 


N 


to hnd a new frame F = 
mutual coherence. 

For alH = 1,..., TV the optimization problem we study is 


that is near (in a sense that is made exact) to the initial one, with smaller 


minimize max Kh\. 

\\f.-h.\\l<Ti 0-, 


For clarity, we can also define Hi = 


( 10 ) 


and thus the objective function of ([T0|) 


hi ... hi-i h.i+1 ... hjq 

becomes \\Hf fi\\oo- Solving this problem, we dehne for each vector hi in the reference frame a trust region (a 
closed TO-ball centered at hi of radius s/Ti) where we search for a new vector fi such that its correlation (in 
absolute value) with the other vectors in the frame Hi is smaller than that of hi. The only input parameter of the 
optimization procedure S K, 0 < <C 1, is a measure of how far we are willing to leave the reference frame 








6 


H in the search for the new frame F. Since now we are dealing with a convex objective function and constraint, 
the formulation in ( fTOl i can be viewed as the convex relaxation of (|^. 

Observe that each vector of the frame can be computed separately in ( [T0| ). Given a hxed frame H assume that fi 
is the only optimization variable and, without loss of generality, that hjhi > 0,1 < j < N (when not, flip the sign 
of hj). We stress this form of the frame since this imposed structure reduces the size of the convex optimization 
problems that need to be solved and, as we will see in the following subsections, highlights several properties of 
frames. 

Problem ( |T0| ) can be separated into N convex optimization problems for i = 1,..., TV 

minimize max (hJ fi) (11) 

\\f.-hi\\l<Ti j; i#* 

This optimization problem lies at the heart of the proposed incoherent frame design method. Since the solution 
will lead to vectors that are not of unit norm (due to the inequality constraint) a normalization step must always 
follow. Solving can be done in polynomial time using, for example, optimization strategies like interior point 
methods ll46l . 

Notice that for a given frame, the distances between its vectors (and consequently its mutual coherence) are 
preserved under column permutations and orthogonal (rotational) transformations. Given a frame H with Gram 
matrix G = H^H, if we rotate the frame using an orthonormal transformation Q G i.e., Q^Q = I, then 

the new frame iTnew = QH has the same Gram matrix since Gnew = = H^Q^QH = H = G. 

Following this observation notice that we can hx the hrst vector of a frame to any arbitrary vector (e.g., ei) since 
we can always compute a rotation Q such that the hrst vector has a predehned, hxed, position. Thus, we can solve 
problem ([n]i for i = 2,..., A^. 

All these properties will be used to simplify the structure of the proposed algorithm. We next proceed to study 
some of the properties of Q- 

A. The choice of parameter Ti 

In this section we discuss the choice of the only parameter needed Ti. Since the original equality constraint 
II/ill 2 = 1 tiot tractable in (|^, we analyze the consequences of the relaxed inequality constraint introduced in 

([n), \\h-K\\i<T,. 

In the optimization problem ( [TT] l we dehne for each vector belonging to the reference frame H a trust region, 
called here Ti, represented by a closed m-ball centered around the reference vector hi of hxed radius ^/Ti- It is 
in this trust region that we search for a new vector fi that replaces the reference hi such that the coherence of 
the frame is reduced. The question is how to choose T such that we can guarantee that the mutual coherence of 
the frame is not increased after the normalization of the solution to ( [TT| l. First, notice that the solution to o will 
always be on the hyperspherical cap, denoted here by Bi, of the trust region % that is bounded by the tangents to 
Ti from the origin. This is true because any solution TgT produces a dot product higher than its projection on 
Bi - a scaled jT G Bi,0 < ■y < 1. This leads to the conclusion that the inequality constraint is satished always 
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Fig. 1. A 2D graphical description of the trust region 71, hashed in grey, centered at hi with fixed radius y/Ti and the boundary Bi depicted 
by the bold arc of 7i where all solutions lie in a typical situation generated by ||/i — /iiHl = Ti. 


with equality \\fi — hi \\2 = Ti. Since Ti > 0, the solution fi is never on the unit m-sphere and thus a normalization 
step must follow - this is due to the fact that {fi \ fi G Bi} n (fi \ fi G Ti and H/ilH = 1} = 0. A graphical 
description of these regions is shown in Figure 

When there is no change in the direction of the new vector of the frame the solution is /i = (1 — \/Ti)hi. 
Otherwise, a change 0 in angle from hi might cause a reduction in coherence with the other vectors of the frame. 
Because the solution belongs to the cap Bi, any deviation 6 from the reference vector leads to a change in the norm 
of the solution 

ll/i|| 2 (^') = cos 6 »- \/cos 2 6 »- l + T, ( 12 ) 

for 0 < 0 < arccos y/l — T. This leads to 1 — ^/T < ||/i|| 2 ( 6 *) < \/l — T. Relation ( fTS] ) follows by the law of 
cosines in the triangle with vertices in the origin, hi and any point fi in the region Bf. T = l + |i/i|| 2 ~ 2 ||/i||| cos 6. 

Remark 1. (The choice of Tj and the convergence of the proposed algorithm): The solution region Bi must 
not contain any vector collinear with the other vectors hj,j T *■ For vector hi if the smallest angle with the 
other vectors is denoted by then we need to have which reads arccos — T < ())™" and hnally 

develops in 

Ti<l- cos^ 0”” = 1 - max \gij (13) 

r, 

Consider that after solving o for a hxed i with the T computed by ( [T3] l we obtain the solution fi which makes 
an angle 9 with (1 — y/Ti)hi. This means that 

\\H^f,Woo <\\Hlh,\\^{l-./fi). (14) 

To finalize the new frame vector the normalization step that places the solution fi on the unit m-sphere follows. 
We would like the normalized solution /i(||/i|| 2 (^*))~^ to still be as good as (or better than) hi, that is 

WH^Moo<WH^h,WooWM2{0), 


(15) 
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This inequality is always true since ( [T4| ) is true and ||/i|l 2 (^) > 1 — Intuitively this makes sense since the 
norm of the solution fi of o increases with the increase in the angle 9, by ( |T^ . Ultimately, this means that the 
solution fi provides a coherence level at least as small as hi and therefore no increase in coherence can happen. The 
proposed algorithm solves problem GD for each vector of the frame and thus the mutual coherence is monotonically 
decreasing with each step solving GD with parameters given by ( |T3] l. Since the coherence is bounded below, the 
convergence of the overall algorithm to a local minimum is guaranteed. ■ 

Remark 2. (ETFs are local minimum points): Given an ETF, any solution of GD for hi that is different from 

/, = (!- VT\)h, causes an increase in the correlations with the other vectors not only because of the decrease 
in the angle between at least one of them but also since any deviation 6 from hi causes a strict increase in ( [T^ . 

Thus, ETFs are local minimum points of G3 (up to the normalization factor). ■ 

The results presented in Remark 1 show the effect that parameter Ti has in the optimization problem and guidelines 
on how to choose it in order to ensure and maximize the decrease in coherence. Next we deal with the overall 
optimization problem. 


B. Analysis of the optimization problem 

In this section we analyze the optimization problem at the heart of the proposed method. Since in the previous 
section we detail the effect of the parameters Ti, we are now mainly concerned with the convex relaxation made 
in the objective function. In this section we analyze optimization problem GI): 


minimize max {hj fi), 
f.-. i; 


which written in a standard form reads 


minimize c x 

x=[fi-, t] 


subject to Ax < 0 

x^Bx — 2b^X + 1 — T) < 0, 


where B = 


I 0 

0^ 0 


'i+l)x(m+l) fjT _ 


hT 0 


e = 


1 

'Jlxm 


e A = 


(16) 


-l(Ar-i)> 


/ i)x(m+i) ^ holds the objective function value. For the solution x G the Karush-Kuhn- 

Tucker (KKT) conditions read: 


Ax < 0, 


Bx — 2b^x + 1 — Ti < Q, 


Xj{Ax)j = 0, Bx — 2b^ X + 1 — Ti) = 0, 


(17) 


c + A^ \ + 2Xn{Bx — b) = 0, 


with j = 1,..., iV — 1 and where A G Ajv G K+ are the Fagrange multipliers. Develop the last equality to 

get 

' H,\ = -2Xn if^-h,), 

N-l 

Xj = 1 and Xj = 0 when hj fi t. 


( 18 ) 
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Multiply both side by ff and notice that H^X = t to reach 




( 19 ) 


Since Aat > 0 and, by ( fT2| ), f^hi - \\fi\\l = ||/z|| 2 COs 6 » - ||/i||| = ||/i|| 2 Vcos 2 6 * - 1 + T) > 0 we have that 
t > 0. Notice that Aat may be nonzero since the associated inequality constraint \\fi — hi \\2 < T) is always met 
with equality (by the argument provided in the previous section). ■ 

Remark 3. (On the existence of real equiangular tight frames); Given an ETF H G define the frame 

Hi = hi ... hi-i hi+i ... hjs! € assuming that for fixed i we have h-fh-j = 7 ^ i, then 

Hf Hi has eigenvector 1 G with eigenvalue {N — m)lm. 

To see this develop Q for v = hi to get 

N / N 


j=t 

and multiply both sides by Hf to finalize 

mp ^ r 


h, = 


V 




TO/r 

N — m 


HX 


Ht Ha = Ht h, => {Ht Hi)l = 


N 


1 . 


( 20 ) 


N — m " " . . . 

We use these results to show how the system of equations in ( fTS] ! behaves when dealing with an ETF. Also, 
result ( |20| ) has several direct consequences for H: 

1) With the assumptions made on hi the Gram matrix is 


G = WH = 


'k 

pi 

k 


1 

pT 

k 

pi 

k 


( 21 ) 


2) Each vector of the frame hj has the same number of positive (N^) and negative (N~) correlations with the 
other vectors in the frame, except for the i* vector for which = TV — 1 and N~ = 0. 

3) With ( | 2 T] i, equality in ( |20l i and the additional positive correlation with hi we obtain (TV^^ — 1 — + 1 = 

{N — m)(m for all hj,j ^ i. Use © and we have 

N — 2m 




1 . 


m/i 


( 22 ) 


The fact that the first term on the right hand side is an integer for ETFs was already shown in 11451 . This result 
offers another possible interpretation of that quantity. Information about the signs in the Gram matrix can be 
used to compute, for example, the average coherence ill for ETFs. This concept was linked to the success 
of sparse approximation algorithms. ■ 

Remark 4. (The KKT conditions for ETFs): Considering Remark 2, when dealing with an ETF, we have that 
fi = {l- VTi)h.i, t={l- VTl)h and thus ([T^ reduces to Aat = fi{2y^) As a consequence in •El we have 
that 


HiX — 


( 23 ) 
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With this result and ( [2Q| > observe that 

h, = = -H,\ (24) 

ly — m 

which leads to A = 1 = G ■ 

Remark 5. (Description of some local minimum points): Let J denote the set of indices j, with j ^ i, for 
which f^hj = t and let Hj be the matrix H constraint to the columns indexed by J. From ( |T8] l it is clear that 
the change ri = hi — fi is in the column space of Hj. 

Consider now a frame for which Hjhi = pil, where pi has the mutual coherence value, and without loss of 
generality that J = {1, ... ,rn}, and thus Hj is of full rank and has empty null space N{Hj) = 0 (we do not 
consider O^xi)- Then consider that we are looking for a direction such that Hj{hi — ri) = ^ 2 ^ and additionally 
h"^_^ihi = /i 2 with /J .2 < /ii - in effect we are looking to find a direction such that the coherence is maximal 
with one extra frame vector. With the given full rank Hj the only possible solution is thus = (/ii — ^ 2 )/ 

This means that the algorithm cannot make any progress. Of course the results holds when H is an ETF and then 
\J\ = N - 1, fii = /X, ^2 = (1 - VTi)h and thus n = y/Tihi. 

Since we are also looking for highly overcomplete frames {N ^ m) this result might be worrying. To overcome 
this drawback, we will introduce a heuristic step in the proposed method that deals with such situations and allows 
the optimization problems to make further progress after getting stuck in such local minimum points. ■ 

Remark 6 . (On the constraint that defines the trust region Ti): In this implementation of ( fTT] i we use an 
£2 constraint to dehne the trust region around each reference frame vector. We could use other trust regions, like 
the ones dehned by the £i or l^o norm. These ’’box” constraints have the extra beneht of keeping the optimization 
problem ( [TOl i to a linear program (as dehned in Q, the objective is already linear). Of course, since the dehnition 
of the trust region is crucial to the choices of Ti and the analysis of the algorithm, the results presented in this and 
the previous sections, does not hold anymore. ■ 

Remark 7. (Reducing the complexity of the optimization problem): We have seen in Remark 5 that is in 
the column space of Hj where J is the set of indices that achieve maximum coherence with fi. For some j, it 
may be that the correlation of hj with hi is so low that given the trust region constraint there is no way this index 
j will belong to the set J. We want to solve •HD only for indices j,j i, that may participate in J. Formally, we 
want to remove from the optimization problem, from Hi, the vectors j for which 

cos{(l)j - < cos(<^“" + (25) 

where the left hand side is the maximum coherence that can be achieved by vector hj with fi, after normalization, 
and the right hand term describes the maximum decrease in mutual coherence that can be achieved, both after 
solving o {(j>j is the angle between hi and hj ,j i, 0 ™"’ = min^ (j>j and is the maximum allowed angle 
change from the reference hi). Together with equation ( [T3| ) that states ( |25] l reduces to 4>j > 

This step proves efficient when the frame is coherent) - for an ETF nothing is dropped. ■ 

The results and the remarks presented in this section come together to construct an efficient method for incoherent 
frame designed which is described in the following section. 
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Algorithm 1 Sequential Iterative Decorrelation by Convex Optimization (SIDCO). Given dimensions (m, N), 

construct frame H G as incoherent as possible until maximum number of iterations AT is reached. 

Initialization: 

1) Create H G with random entries from a Gaussian distribution. Normalize its columns. 

2) With H = update the frame by the unit polar decomposition (by Theorem 2 of ^QM \. H = UV"^. 

3) Normalize the columns of H. Set reference frame H. 

Iterations fc = 1,..., iT: 

1) For each i in randomized {2,..., N}: 

a) Flip the sign of hj where hf hj < 0 for all je iV}\{i}. 

b) Compute radius Ti of trust region % by ([T3J. 

c) Define J = {j \ such that hjhj is maximum}. 

If |J| < m reduce problem dimension by ( |25] l and solve optimization problem (fTT) for fi. 

d) Normalize and update hi = /i||/i ||2 

2) If the algorithm has converged, update frame by its unit polar decomposition H = UV^ (again by Theorem 
2 of 1^) and normalize columns. 


The proposed method, called Sequential Iterative Decorrelation by Convex Optimization (SIDCO) gradually 
decreases the coherence of an initial frame by finding a new frame that is close to the previous one and achieves 
lower coherence. 

The general structure of SIDCO is represented by the following iterative optimization procedure that updates 
each vector of the frame in the sequence one at a time. The method brings together the results presented in the 
previous sections. 

Again, we stress that this optimization procedure works by directly modifying the frame and not its Gram matrix. 
The SIDCO approach is somewhat similar to the sequential convex programming framework that approximates 
solutions to nonconvex optimization problems by solving a series of locally convex optimization problems in a 
defined trust region. 

The initiahzation. The initial frame we consider has its entries drawn randomly from a Gaussian distribution. 
This simple initialization is useful in this case since it has been shown that random frames have very low coherence 
values and actually, as N tends to infinity, they converge to tight frames ll48ll . Vectors from a Gaussian distribution 
are uniformly placed on the unit m-sphere since the Gaussian distribution is rotational invariant. By Theorem 2 
of ll24l . we next perform a polar decomposition of the frame and we reconstruct using only the unit part of the 
decomposition. To finalize, normalize the frame vectors. Herein, the initial reference frame is Hq. 

The iterations. By the argument provided in the previous sections, at each iteration of SIDCO we solve a 
sequence of iV — 1 convex optimization problems that produce the vectors of a new frame such that the mutual 
coherence is decreased. The frame vectors are chosen in a randomized way so to further avoid the convergence in 
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a local minimum point. Before solving the optimization problem, several choices need to be made. A discussion 
of these follows. 

First, we change the orientation of the frame vectors such that the current variable vector has positive correlation 
with all the other fixed vectors. This allows for both a smaller optimization problem and a simplified analysis of the 
optimization problem We pick the maximum size of the trust region Ti which also guarantees that the mutual 
coherence of the frame will not increase after solving the optimization problem ( [TT| l, as discussed in Remark 1. 
We check if the current frame vector hi can be improved and if so we reduce the dimension of the optimization 
problem, by the arguments provided in Remark 5 and Remark 7, respectively. We finalize the new vector by a 
normalization step placing it on the unit m-sphere. 

Since Remark 5 states that the algorithm does not update a frame vector when it reaches the maximal correlation 
with a number of m other frame vectors, we add a heuristic step to push the current frame from its local minimum 
if the algorithm has converged too quickly. We say that the algorithm has converged if in the last 3 iterations the 
average progress in mutual coherence is less than estop = 10“^. The step applies a polar decomposition as described 
by Theorem 2 of ll2^ . We expect that this update will lead to a relatively small temporary increase in the mutual 
coherence that will be reduced in the following regular iterations, hopefully to levels lower than the one before 
the application of this step. As we will show experimentally, SIDCO makes significant progress in the reduction 
of the mutual coherence in a relative small number of iterations (and thus quickly reaches a local minimum point 
as shown in Remark 1). Therefore, the heuristic step is important in order to allow further progress and push the 
coherence as low as possible in the maximum of K iterations. As discussed in Remark 1 the mutual coherence 
is reduced, or kept at the same level, with every step solving O- Clearly, the application of Theorem 2 of ll24ll 
breaks the monotonicity and allows for an increase in coherence. Still, in many applications (for example El) 
where incoherent frames are desired there is an underlying assumption that the frame is also tight. In general, GFs 
are not tight frames. 

Obviously, solving a sequence of such problems does not guarantee that we reach a maximally incoherent frame 
(GF or ETF) but considering that we actually solve a sequence of minimization problems we expect to have 
many components of the objective function equal in magnitude to the i^o norm of the correlations between the 
current vector fi with the others, and thus to be close to such a frame. 

An extra advantage of this formulation is that, according to the design needs, it is very easy to add linear equality 
and convex inequality constraints. We use this fact in the results section to construct incoherent frames that also 
satisfy other convex constraints (like nonnegativity). 

IV. Results 

In this section we show the capabilities of SIDCO to: provide incoherence frames and sensing matrices for 
optimized compressed sensing. Finally, we also make a note on the creation of ETFs. 


TABLE I 

Average incoherence results reached by SIDCO eor einal erames H e with increasing values oe N for 100 

RUNS. The initial frames Hq are also shown together with the Welch bound (WB) for perspective. 
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N 

avg(/i(ffo)) 



WB 

a.vg{n.{H)) 

30 

0.7264 

0.2057 

0.2073 

0.1857 

0.1841 

45 

0.7429 

0.2523 

0.2602 

0.2132 

0.1941 

60 

0.7554 

0.2808 

0.2866 

0.2255 

0.1994 

75 

0.7654 

0.3024 

0.3072 

0.2325 

0.2028 

90 

0.7736 

0.3196 

0.3230 

0.2370 

0.2051 

105 

0.7806 

0.3367 

0.3395 

0.2402 

0.2067 

120 

0.7867 

0.3502 

0.3585 

0.2425 

0.2077 

135 

0.7922 

0.3661 

0.3715 

0.2443 

0.2083 

150 

0.7969 

0.3790 

0.3830 

0.2458 

0.2086 


A. Designing incoherent frames 

Given dimensions (m, N) the first task is to use SIDCO in order to create frames as incoherent as possible. 

Tables 00 and m show the design capabilities of SIDCO for numerous frame dimensions, either for fixed 
dimension m and increasing N or vice-versa. In each case, we run SIDCO with 100 different initializations. We 
show the mean mutual coherence of the initial frame, the minimum mutual coherence and the average mutual 
coherence reached by SIDCO in the 100 runs. In each run, we compute the average mutual coherence defined as 
fi{H) = (iiot to be confused with the average coherence i/ of HtI I. For perspective 

we also show the Welch bound (WB). The coherence levels reached are very close to this bound. In the initial tests. 
Tables 00 and 1 ^ we design frames that are at least 2-overcomplete {m, 2m) and maximally 10-overcomplete 
(to, IOto). As expected, for larger N the gap to the bound increases but is still kept in reasonable limits. Furthermore, 
notice that the mean mutual coherence reached is close to the minimum coherence, showing that the proposed method 
is not very sensitive to the choice of the initial frame. SIDCO runs for K — 200 iterations. 

The hrst comparison we make is between IDCO ll2Tl and SIDCO. Table |IV] compares the two for the design 
of highly overcomplete frames (up to TO-overcomplete) for dimension to = 64. The IDCO results are lifted from 
El. We also show the Welch bound and [sj from the maximum sparsity level of the signals that can be 
successfully recovered in each case by solving an £i minimization problem with the SIDCO frame. Both methods 
run for K = 150 iterations. Notice that the results of the two methods are close for small frames while the 
performance gap increases with N in favor of SIDCO. In comparing these two methods, the big advantage of 
SIDCO is in terms of speed. For example, IDCO produces the (64,1280) frame in an over-week run while SIDCO 
terminates in an over-night run - and actually with IDCO it is not computationally feasible to proceed for to > 1280. 
The fact that each frame vector is updated individually means that SIDCO does not require large memory resources 
and the fact that the parameters Ti are chosen to maximize performance (and not heuristically like for IDCO) 
explains the difference in performance. To show the versatility of the proposed method, SIDCO is used to produce 
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TABLE II 

Analogous to TableIiIfor final frames H e 


N 

avg(/i(ffo)) 



WB 

avg(^j{if)) 

50 

0.6343 

0.1584 

0.1599 

0.1429 

0.1420 

75 

0.6491 

0.1939 

0.1954 

0.1644 

0.1499 

100 

0.6600 

0.2164 

0.2182 

0.1741 

0.1541 

120 

0.6937 

0.2303 

0.2378 

0.1787 

0.1568 

125 

0.6690 

0.2339 

0.2405 

0.1796 

0.1585 

150 

0.6766 

0.2476 

0.2491 

0.1832 

0.1598 

175 

0.6830 

0.2589 

0.2640 

0.1857 

0.1607 

200 

0.6887 

0.2695 

0.2763 

0.1876 

0.1613 

225 

0.6937 

0.2786 

0.2799 

0.1890 

0.1618 

250 

0.6982 

0.2874 

0.2898 

0.1901 

0.1621 


TABLE III 

Analogous to TABLE[i]FOR final frames H e r™x12o 


m 

avg(M(T/o)) 


a . vg ( p { H )) 

WB 

avg(/i(ff)) 

15 

0.8233 

0.3502 

0.3650 

0.2425 

0.2138 

20 

0.7872 

0.2775 

0.2790 

0.2050 

0.1995 

25 

0.6937 

0.2303 

0.2463 

0.1787 

0.1879 

30 

0.6469 

0.1977 

0.1989 

0.1588 

0.1780 

35 

0.6080 

0.1727 

0.1757 

0.1429 

0.1695 

40 

0.5914 

0.1529 

0.1593 

0.1296 

0.1620 

45 

0.5766 

0.1371 

0.1414 

0.1183 

0.1552 

50 

0.5632 

0.1236 

0.1254 

0.1085 

0.1491 

55 

0.5509 

0.1120 

0.1132 

0.0997 

0.1435 

60 

0.5394 

0.1019 

0.1029 

0.0917 

0.1382 


nonnegative frames of the same dimensions G (all k^j > 0). Because SIDCO deals directly with the 

frame (and not its Gram matrix) the nonnegativy constraints are trivial to add. In this particular setting we do not 
use step 2b) of SIDCO, that is not able in general to keep the entries nonnegative. We are not aware of another 
method that successfully designs frames with such a high degree of redundancy and/or nonnegative entries. 

In compressed sensing, frames are usually called incoherent if the mutual coherence obeys ii{H) < 
the asymptotic coherence bound (|^ for large N and the mutual coherence achieved by algebraically constructed 
ETFs. For all dimensions where examples are presented this bound is reached only for a small number of possible 
redundancies. In all the cases (m = 15, to = 25 and to = 64) only the frames maximally 3-overcomplete obey this 


bound, with values 0.2582, 0.2 and 0.125 respectively. The same holds for the results in Table III where N = 120 
for TO > 40. 


We now give some insights into the way SIDCO reaches highly incoherent frames and comparisons with 
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TABLE IV 

Incoherence results reached by IDCO and SIDCO for final frames H e with increasing values of N . The 

INITIAL frames Hq ARE ALSO SHOWN TOGETHER WITH THE WELCH BOUND (WB) FOR PERSPECTIVE. THE LAST COLUMN SHOWS THE 
coherence of the NONNEGATIVE ERAMES H+ S DESIGNED BY SIDCO. 


N 

KHo) 

m{-^J)idco 

SIDCO 

WB 

L^-J 


128 

0.3606 

0.0987 

0.0979 

0.0887 

5 

0.2480 

192 

0.4596 

0.1206 

0.1186 

0.1023 

4 

0.2836 

256 

0.4176 

0.I37I 

0.1346 

0.1085 

4 

0.3058 

320 

0.4755 

0.1451 

0.1429 

0.1120 

3 

0.3217 

384 

0.5395 

0.1536 

0.1510 

0.1143 

3 

0.3369 

448 

0.4993 

0.1599 

0.1580 

0.1159 

3 

0.3455 

512 

0.5040 

0.1634 

0.1615 

0.1170 

3 

0.3539 

576 

0.5009 

0.1720 

0.1693 

0.1180 

3 

0.3592 

640 

0.5469 

0.1764 

0.1739 

0.1187 

3 

0.3675 

768 

0.5250 

0.1894 

0.1854 

0.1198 

3 

0.3744 

960 

0.6023 

0.2136 

0.1979 

0.1208 

3 

0.3878 

1280 

0.6386 

0.2256 

0.2115 

0.1219 

2 

0.4042 

1600 

0.6678 

- 

0.2230 

0.1225 

2 

0.4162 

1920 

0.6890 

- 

0.2317 

0.1229 

2 

0.4272 

2240 

0.6934 

- 

0.2394 

0.1232 

2 

0.4355 

2880 

0.7001 

- 

0.2511 

0.1236 

2 

0.4476 

3520 

0.7711 

- 

0.2604 

0.1239 

2 

0.4514 

4096 

0.8820 

- 

0.2617 

0.1240 

2 

0.4603 


competing methods. First, notice in Figure the sorted unique correlations (in absolute value) between distinct 
vectors of the frame H S r64x 128 designed with SIDCO. The figure shows all 8128, N{N — l)/2 for N = 128, 
unique correlations together with the mutual coherence reached /i(ff)siDCO = 0.0979 and the Welch bound of 
0.0887. We report that 66.14% of the correlations are above 0.99/i(iT)siDCO and 74.21% are above 0.95/i(ff)siDCO- 
The figure shows how most of the correlations are close to the achieved mutual coherence. 

Figure shows the evolution, by iteration, of several algorithms for the design of frame H G SIDCO 

outperforms all methods in the literature. By the 5* iteration SIDCO achieves the mutual coherence that will 
be finally achieved by the third best approach iol. The second best approach is IDCO. This plot highlights the 
difference between IDCO and SIDCO not just in terms of the final result but also, empirically, in terms of the 
rate of convergence. The difference originates in the way the parameters are chosen: heuristically for IDCO and 
by Remark 1 for SIDCO. This shows the effectiveness of the method. Also, for SIDCO the mutual coherence is 
decreasing at each iteration. Exceptions make a few iterations where spikes in the mutual coherence are recorded. 
These are the times when heuristic step 2b) is applied. The first time step 2b) is applied is at iteration number 62 
where the mutual coherence is 0.3620. Until the 200* iteration step 2b) is applied 5 more times and the final mutual 
coherence reaches /i(ff )sidco = 0.3502. Figure is analogous to the previous figure but this time the frame in 
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Fig. 2. The unique absolute value correlations of H £ The figure shows their distribution and that many of them reach, or are 

close to, the mutual coherence value of the frame. 



Fig. 3. Evolution by iteration of several algorithms for incoherent design of one frame H G The initial frames are the same for all 

algorithms. 

question is H G k 25 x 120 Again, the performance of BOll is reached in only 6 iterations while IDCO converges at 
a lower rate than SIDCO. In this situation the heuristic step 2b) is applied a total of 8 times: the hrst time SIDCO 
converges to a value of 0.2409 in 58 iterations and hnally it reaches /x(ir)siDCO = 0.2303. 

The SIDCO approach clearly outperforms all previous methods, in the hrst case quite signihcantly. In both 
these situations, SIDCO runs for 200 iterations and the other methods for 1000 iterations - the result of SIDCO is 
extended to ht the plot. Both cases show that in the hrst iterations SIDCO makes signihcant progress while the later 
iterations provide additional progress but at a much lower pace. Running SIDCO for a large number of iterations 
will in general provide lower coherence levels. Taking this into account, in practice, if there is a time limit, SIDCO 
may run for only a few iterations. In these experimental runs we have used the generic convex optimization solver 
CVX 149)1 - it was not in the scope of this paper to build a custom solver. One may also argue that SIDCO needs 
to run only for a few iterations and that step 2b) can be ultimately avoided. This is due to the fact that SIDCO 
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Fig. 4. Evolution by iteration of several algorithms for incoherent design of one frame H £ JJ25X120 jjjg initial frames are the same for all 
algorithms. 



Fig. 5. Mutual coherence of several frames H £ ]^rn.xl20 designed by various algorithms. SIDCO is represented here by the minimum 
coherence column from Table [hI] Results shown are averages obtained over 100 realizations. All algorithms are initializated with the same 
frames. 


achieves low coherence very quickly, only after a few iterations, and then the progress due to step 2b) may be 
considered too little to be worth the large number of iterations that follow. Still, in a situation with no pressing 
time limit, SIDCO should run for a number of maximum iterations K as large as possible. 

Finally, Figures and compare the SIDCO method with the best competing methods in the literature and show 
again how it outperforms all of them. The difference in performance depends on the redundancy of the frames. In 
Figure for highly redundant frames SIDCO outperforms the other methods by a signihcant margin (see m < 25) 
while otherwise the gap is diminished. Also, notice that for the frames with low redundancy (see m > 35) SIDCO’s 
mutual coherence approaches the Welch bound so closely that no room for great improvement exists. In Figure |^we 
see how SIDCO fares against IDCO for frames H G We also show the Xj^/m = 0.1581 bound that 

















18 



Fig. 6. Comparison between IDCO 1211 and SIDCO for frames H £ . We also highlight the ll^/m = 0.1581 bound. Both algorithms 

are initialized with the same frames. SIDCO outperforms IDCO on average by 10%. 


is met again by frames at most 3-overcomplete. Since SIDCO is an improvement over IDCO there is no surprise 
that it outperforms it in all cases, the gap increasing with redundancy. In this situation, the heuristic step 2b) plays 
an important role in outperforming IDCO (to match the number of iterations). 

For all frames we also compute the frame potential (|^. On average this value is within 1% of its minimum value 

(N'^/m). 

B. On the design of equiangular tight frames 

We also applied SIDCO to the design of real ETFs. Although SIDCO performs very well in all previous examples, 
and substantially outperforms previous methods, in this case it fails to achieve the exact structure of an ETF We 
run SIDCO for the pairs (m, N) with N < 100 for which we know real ETEs exist ifT^ and it is virtually never 
able to exactly reach (with a coherence error of 10“®) the said ETE. This is even when in most cases the coherence 
is below the 1/a/to bound. It seems these iterative approaches, without explicitly considering structural properties 
of ETEs (Cl, ES and Remark 3), are not generally able to construct them. Note that we are never interested 
in the, trivial, real ETEs (to, to) and (to,to+ 1) that exist for every to > 1 and are easily constructed ifT^ . The 
difficulty of numerically constructing ETEs was also pointed out in ll24l . 

C. Optimized compressed sensing 

We now move to show the efficiency of the constructed frames for compressed sensing El. At the heart of 
compressed sensing we hnd the underdetermined system of linear equations y = Fx, where F G is called 

the sensing matrix and where we know that x has a sparse representation like x = Da where a G is s-sparse 
in a known dictionary D G The goal is to recover a from a number of measurements to as low as possible 

using a recovery algorithm. The correct recovery of a depends on the number of measurements to, the coherence 
of the sensing matrix F and the sparsity level of a. We generate a random Gaussian dictionary D and random 
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Fig. 7. Relative reconstruction errors for sensing matrices with m measurements and fixed sparsity s = 4 and fixed dimension TV = 80 for 
various algorithms. The reference performance is the random sensing matrix. For SIDCO the sensing frames have, in increasing order of m, 
the following coherences: 0.3067, 0.2416, 0.1980, 0.1690 and 0.1439. 


s-sparse representations a (with a random support) which we then try to recover using different sensing matrices 
(random or frames produces by various algorithms, including SIDCO). We show the results in synthetic experiments 
for N = 80, M = 120 and various values of m and s. The sparse recovery is done using the Orthogonal Matching 
Pursuit (OMP) algorithm and the performance indicator is the relative energy content of the reconstructed signal 
arec as compared with the original; ||a — areclli/ll^lli- choice of dimensions, we average the recovery 

results over 10® runs are we present them when we vary the number of measurements m for fixed sparsity s = 4 
and fixed dimension N = 80. Results are shown in Figure [7] Like in the previous section, SIDCO is better than 
every other method presented with the performance gap increasing for larger number of measurements. 

As others point out Ho), the performance gain of using incoherent frames as sensing matrices does not scale as 
much as with the improvement in the mutual coherence. Observe that in Figure]^ for m S {15, 20} all results are 
very close to the random reference - whereas the values of the mutual coherence of the initial, random, frames Hq 
in Tables m [In] and [lyi are very high compared to the mutual coherence achieved by SIDCO. To see why this 
might be the case we remind the reader that compressed sensing is able to recover the correct support when the 
amplitudes of the non-zero entries of a are above a constant times \jMlm log M ll50ll . which takes high values 
for relative small m and that has been shown to hold for Gaussian random sensing matrices with order s log M 
measurements M- Apart from the mutual coherence, recent developments m show that unit norm tight frames 
perform well in compressive sensing applications when measuring the reconstruction average mean squared error. 


D. Incoherent dictionaries 

Observe that the construction of the incoherent frames is completely determined by the choice of dimensions 
{m,N). These processes are blind to the nature of the data which will be represented in the frames. The idea in 
this section is to adapt the highly incoherent frames previously created to a given data sample such to reduce the 
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Fig. 8. Relative reconstruction errors of several dictionaries F E using the training procedure described in this section starting from 

frames Fq previously designed by SIDCO. We consider two sparsity levels s £ {4,12}. We also show in dotted lines the reconstruction errors 
for initial frames Fq. 


reconstruction error. 

We are presented with a sample dataset Y G and the goal is to adapt a given incoherent frame 

designed by SIDCO, such that the reconstruction error of the provided dataset is minimized. We use the word 
’’adapt” because we are interested only in transformations that preserve the incoherence of the frame i^g- As such, 
we are interested in finding an orthogonal transformation Q such that F = QFq minimizes || y—i^X||i;’||Y||p^ (the 
relative reconstruction error) where the sparse representations X G are found via some sparse approximation 

algorithm, like for example the Orthogonal Matching Pursuit (OMP) ||7]. 

Consider the optimization procedure that at step i, given a fixed frame and sparse representations Xi_i, 
solves: 

1) Find Qi via the orthonormal Procrustes problem ll52ll : 

minimize \\Y - QiF^^iXi^iWp, (26) 

Qz, QiQj'=QfQz=l 

with solution Qi = UV^ from YXj'_j^F^_^ = U'EV'^. 

2) Update the dictionary F} = QiFi-i. 

3) Construct the new representations Xi using Fi and the OMP algorithm with target sparsity s on the dataset 

y. 

The final solution is the accumulation of the rotations, say after K steps, into Fk — Qk ■ ■ ■ Q 2 Q 1 F 0 which 
preserves mutual coherence, i.e., ^.{Fk) = /r(Fb)- For simplicity, herein we denote F = Fp. This optimization 
procedure was used in ll53l together with an iterative projections approach to design incoherent overcomplete 
dictionaries and in m as the basis for an initialization strategy for general dictionary learning. In this section we 
propose to use the highly incoherent frames created via SIDCO and then use the fast rotational transformations to 
adapt it to a given dataset. This way, the coherence of the dictionary is fixed during the whole training procedure 
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while the reconstruction error is reduced. 

We use image data from popular test images (Lena, peppers, etc.) by extracting non-overlapping blocks of 
8x8 pixels from which we remove the means and we subsequently normalize. The overall training matrix is 
Y G k 64 x 16384 initial frames we use the incoherent frames F G K 64 xtv previously created and run the training 
procedure based on rotations for K = 500 iterations. We show in Figure]^ the relative construction errors achieved 
by the incoherent frames before, Fq, and after training, F. The improvement is obvious but notice that it does not 
scale with dimension N due to the incoherent structure. 

V. Conclusions 

In this paper we present a method for the design of highly incoherent frames that operates directly on the frame 
vectors and is based on solving a series of convex optimization problems that approximate the original nonconvex 
design problem. We derive conditions under which the algorithm converges and show that equiangular tight frames 
are in fact local minimum of this method. Extended simulations, covering low and high dimensional spaces and 
different levels of redundancy, show that the proposed method substantially outperforms all previously proposed 
incoherent frame design methods. 
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